Device and method for synchronising broadcast audio-visual programmes and complementary data

ABSTRACT

The invention relates to a device and a process for the synchronization of broadcast audiovisual programs and of complementary information.  
     The device comprises a pictures and/or sound detection set ( 7 ), capable of extracting at least one semantic element ( 41 - 43 ) from the content of the audiovisual programs currently being broadcast. It also comprises a recognition unit ( 8 ) for recognizing these semantic elements, which is linked to a program guide ( 20 ) which comprises one or more chronologically ordered lists ( 21 - 23 ) of information sets respectively associated with audiovisual programs. The recognition unit selects for at least one of the programs currently being broadcast, the information set which is most probably correlated with these semantic elements. It performs a prior learning enabling it to select the appropriate information sets on the basis of the semantic elements. A synthesis block ( 5 ) synchronizes the audiovisual programs with the selected information sets.

[0001] The present invention relates to a device and to a process forthe synchronization of broadcast audiovisual programs and ofcomplementary information, as well as to corresponding applications.

[0002] In interactive television systems, a central server receives fromadvisory centers a program guide which contains information relating tothe programs which are to be broadcast. A television viewer can thus,from his television, directly access the information of interest to him.

[0003] A particularly useful and user-friendly system would offer theviewer the possibility of having information relating to the programcurrently being broadcast displayed live on his screen. This display(superimposed on the picture or spatially offset) would then beautomatic, and would last for example for a few seconds, or would beactivated by the user.

[0004] However, such a system would require synchronization between theprogram currently being broadcast and the display of the complementaryinformation. Now, usually, the initially intended schedules included inthe program guides are purely indicative: it is common for the programsnot to start and finish without delays of a few minutes or more withrespect to these schedules. It even happens that some programs arewithdrawn at the last moment, for example when a station chooses tobroadcast the end of a sports event until well after the envisaged endschedule, to the detriment of the next transmission. It is therefore notpossible to rely on the schedule indications provided by the programguides in order to hope to match up the broadcast audiovisual programsand the displaying of corresponding complementary information.

[0005] Of course, at the present time some stations send signals makingit possible to identify the program currently being broadcast, but mostof them do not have recourse to such a technique, and the programsconcerned are of very minority appeal.

[0006] The present invention proposes a device and a process for thesynchronization of broadcast audiovisual programs and of complementaryinformation, permitting in particular synchronized real-time displayingof the programs and of the complementary information.

[0007] The device and the process of the invention also make possibleaudible broadcasting of the complementary information (for example forthe hard of hearing), which can interrupt the audible broadcasting of aprogram for a short instant or, optionally, be superimposed thereon witha higher intensity.

[0008] The device and the process of the invention also haveapplications based on the use of complementary information for purposesother than broadcasting, in particular in respect of automaticrecordings on video recorders. Other applications relate to thesubstitution of content of audiovisual programs. Thus, the user can, forexample, decide in advance an automatic and real-time replacement on thescreen of a certain category of programs (for example transmissions ofvariety shows) with another (for example fictional films), by means of aselection from several broadcasting sources (several stations, internet,local storage system, etc.).

[0009] To this end, the invention relates to a device for thesynchronization of broadcast audiovisual programs and of complementaryinformation in accordance with claim 1.

[0010] The expression “broadcasting channel” is understood to mean anypath or set of paths for communicating information originating frombroadcasting means and destined for reception means, corresponding to agiven broadcast programming. Moreover, the broadcasting of theaudiovisual programs may equally well be performed on the basis of anytechnique, such as network for broadcasting over the airwaves,satellite, cable, internet or local storage system.

[0011] The information set relating to each program contains significantinformation pertaining to this program, such as one or more of thefollowing items of information:

[0012] provisional start and end schedules,

[0013] provisional duration,

[0014] title and possibly subtitle,

[0015] summary,

[0016] parental code,

[0017] format,

[0018] sound quality,

[0019] topic and possibly subtopic,

[0020] linguistic version (O.V. or F.V.) etc.,

[0021] this information being used preferably by the recognition unitduring the operations of correlation with the semantic elementsextracted from the pictures and/or from the sounds. An information setalso comprises other details, useful for knowledge of the program (hencefor example for real-time display) but of little consequence for therecognition of the semantic elements and hence preferably not used forcorrelation, for example:

[0022] director, screenplay writer, composer,

[0023] list of actors and respective roles,

[0024] year of production,

[0025] country of origin, etc.

[0026] As far as the semantic elements extracted are concerned, they areelements which are significant with regard to recognition, that is tosay for matching these semantic elements with the information used.Thus, they advantageously comprise:

[0027] a family of shapes (for example a ball in motion to identify afootball match, middle distance shots of busts to identify debates orpresenters, etc.),

[0028] a sound level (for example a higher sound level to identifyadvertisements),

[0029] a pace of motion (for example a fast pace to identifyadvertisements and a slow one to identify a studio debate),

[0030] a family of visual signs (for example a sign of parentalauthorization to identify the type of film broadcast) or of audiblesigns (for example a violin tone to identify certain generic ones) etc.

[0031] Picture and/or sound detection and recognition of semanticelements is implemented by means of techniques which are in themselvesknown. In particular, the correlation of the semantic elements with theavailable information advantageously uses artificial intelligencemethods constructed for example around neural networks (associationbetween artificial intelligence engine and auto-learning). The device ofthe invention nevertheless imposes on these recognition techniques theconstraint of choosing from an ordered list, this amounting basically topositioning a pointer in this list.

[0032] This constraint relating to the lists of the program guide makesrecognition possible, since it imposes a framework limited to the domainof analysis.

[0033] Moreover, the capacity for recognition is acquired by means oflearning, and is not programmed a priori. This learning allows thesystem every flexibility and permits the consideration of a large numberof parameters related to the picture and to the sound of the programsbroadcast (semantic elements) and to the information utilized(information set). Thus, despite all the uncertainties related to thisor that particular parameter, the parameters as a whole set can providereliable results. The latter are manifested by the most significantprobability with regard to the positioning of the pointer in each list,which corresponds to the strongest correlation between the semanticelements and the information sets in the list.

[0034] The invention is thus based on the combination of three nestedfunctions:

[0035] picture and/or sound detection leading to the production ofsemantic elements,

[0036] recognition of the semantic elements,

[0037] and selection of information from a chronologically ordered listby virtue of the recognition of the semantic elements and underconstraint of an order relation (chronology of the programs).

[0038] An important advantage of the invention is that it does not needany particular means to be implemented at transmission level, it beingpossible for the identification of the appropriate information to becarried out entirely on reception, whether the latter be final(terminals, such as interactive televisions) or intermediate (servers).

[0039] The device according to the invention is preferably locatedgeographically in a central server, intended for making the detailsobtained available to the reception means. The central server thenadvantageously performs in a systematic manner the synchronization withregard to a selection of stations, and transmits the information to theusers automatically (for example on each change of station for a fewseconds) and/or on request.

[0040] Preferably, the synchronization device of the invention is inaccordance with claim 2, and advantageously with claim 3. Thus, thepositioning of the pointer in each of the ordered lists is necessarilycarried out in one direction only, towards a rank higher than or equalto the current position of the pointer. This amounts to forcing thechoice of the elements relating to the current program or to a laterprogram—except in the case when the list is updated. This additionalconstraint eases the recognition operations, at the cost of anadditional restriction with regard to the determination of the relevantinformation sets.

[0041] The initial positioning of the pointer in the list isadvantageously obtained by means of schedules indicated in the programguide and of the current time.

[0042] Moreover, the synthesis block advantageously takes account of theinformation set selected last for updating the indicator. This schememay allow optimal positioning of the indicators over time, and hence mayenable the determination of the current indicator to be made morecertain and more effective. However, it implies that the updating isdone continuously. Moreover, should there be an error in the automaticselecting of an indicator, there is a risk of this error having aknock-on effect on the later choices through lower effectiveness orthrough new errors. In another mode of updating, the synthesis blockuses the schedules indicated in the program guide for a defaultpositioning. The two forms of updating are advantageously combined, theindicator of each list preferably having its position modified wheneverthe updating by schedule or by selection leads to a position later thanthe current position in the list—this modification does not affect theinformation set currently in use, but only the recognition operations.

[0043] The synchronization device is for example implementedperiodically, in time slots close to the start and end times indicatedin the program guides and/or at the request of users.

[0044] In another form of selection of the information set, this orderrelation constraint is not imposed. A reversal of programs is thus madepossible, this occurring perhaps for example in the presence of sportsprograms broadcast live.

[0045] In yet other forms of selection of the information sets, use isalso made of the indicators of the current information sets, but withother types of constraints. For example, backward movements through thelists are permitted, but limited to one or two units and/or forwardjumps are permitted (at least two units), but restricting them to two orthree units.

[0046] Advantageous embodiments of the device of the invention aredefined in the dependant claims 4 to 7.

[0047] The invention also relates to applications of the device of theinvention, such as those stated in claims 8 to 10.

[0048] It also pertains to a process for the synchronization ofbroadcast audiovisual programs and of complementary information, such asis defined in claim 11.

[0049] The invention will be better understood and illustrated by theembodiments and modes of implementation set forth hereinbelow, which arein no way limiting, with reference to the appended figures in which:

[0050]FIG. 1 is a schematic diagram showing a synchronization deviceaccording to the invention, in its environment;

[0051]FIG. 2 represents a program guide used by the device of FIG. 1,

[0052] and FIG. 3 illustrates the basic manner of operation of thepointing of one of the indicators used in the device of FIG. 1, to thecorresponding list of the program guide of FIG. 2.

[0053] The central unit 2 (FIG. 1) of a receiver of audiovisual programs10 is intended to receive the content of the programs 10 originatingfrom a broadcasting set 1 (for example a radiobroadcaster) and to haveaccess to a program guide 20 of an advisory center 3 (for example adedicated service provider). The central unit 2 is then able to produceon the basis of the programs 10 and of information from the advisorycenter 3, audiovisual programs 15 broadcast with synchronizedcomplementary information, and to communicate these programs 15 to meansof broadcasting 4 on the screen.

[0054] The central unit 2 more precisely comprises:

[0055] a synthesis block 5, responsible for controlling all theoperations and in particular for synchronizing the programs 10 withselected information,

[0056] a memory 6, designed to store information relating to the programguide 20 and for the extraction of appropriate information from thisguide 20,

[0057] a picture and/or sound detection set 7, capable of extracting oneor more semantic elements 41-43 (family of shapes, sound level, pace ofmotion, etc.) from the content of the programs 10 currently beingbroadcast,

[0058] and a recognition unit 8 for recognizing semantic elements, whichis designed to match up the semantic elements 41-43 identified andinformation emanating from the program guide 20.

[0059] The recognition unit 8 operates on the basis of chronologicallyordered lists 21, 22 and 23 (FIG. 2) of information sets (schedules,duration, title, summary, etc.) respectively referenced 21 ₁-21 _(n1),22 ₁-22 _(n2) and 23 ₁-23 _(n3), provided by the program guide 20. Eachof the lists 21 to 23 is associated with a broadcasting channel(television station) and its information sets 21 ₁-21 _(n1), 22 ₁-22_(n2) and 23 ₁-23 _(n3) relate respectively to audiovisual programs 10broadcast on this channel.

[0060] Since a program 11, 12 or 13 is currently being broadcast on oneof the broadcasting channels, associated respectively with the list 21,22 or 23 of the program guide 20, the recognition unit 8 is capable ofselecting respectively the information set 44, 45 or 46 of this listwhich is the one most probably correlated with the semantic elements 41,42 or 43 extracted from this program by the detection set 7. Thisrecognition unit 8 is intended for performing a prior learning enablingit, by means of learning materials comprising semantic elements andassociated information, to make the most judicious selections. Thislearning is advantageously continued during use. For selection withprior learning, the recognition unit 8 implements for example neuralnetwork techniques associated with an artificial intelligence engine.

[0061] In the embodiment represented, an additional constraint involvesindicators 31, 32 and 33 (FIGS. 1 and 3) pointing respectively to thecurrent information sets selected from the lists 21, 22 and 23. Theseindicators 31 to 33, stored in the memory 6, give the information setscorresponding to the audiovisual programs broadcast last respectively oneach of the broadcasting channels (for example, the indicator 31 pointsto the set 21 ₂ which corresponds to the program broadcast last on thefirst station—FIG. 3).

[0062] The additional constraint imposed by the recognition unit 8 isfor example that the information set to be selected from one of thelists 21, 22 or 23 is situated downstream of the information set pointedat by the indicator 31, 32 or 33. The chronological nature of the listsis thus used to succeed in identifying the appropriate information morecertainly and/or more simply. In a variant embodiment, the selecting ofa downstream information set is permitted, but provided that the latteris not more than two units away from the current one.

[0063] The synthesis block 5 is responsible for updating the indicators31 to 33 in the memory 6, preferably as a function of the informationsets 44 to 46 selected by the recognition unit 8. This updating allowseffective application of the constraints imposed on the movements of theindicators 31 to 33.

[0064] During operation, the following is undertaken for implementationof the synchronization device as applied to a broadcasting channel (forexample the first):

[0065] the semantic elements 41 of the program 11 currently beingbroadcast on the first channel are extracted by means of the detectionset 7;

[0066] the information set 44 most probably correlated with the semanticelements 41 and complying with the chosen constraint regarding themovement of the indicator 31 associated with the list 21 is selectedfrom the chronologically ordered list 21, associated with the firstchannel, of information sets by means of the recognition unit 8;

[0067] the indicator 31 is updated by means of the synthesis block 5;

[0068] and the program 11 is synchronized with the selected informationset 44 by means of the synthesis block 5.

1. A device for the synchronization of broadcast audiovisual programsand of complementary information, comprising: a picture and/or sounddetection set (7), capable of extracting at least one semantic element(41-43) from the content of said audiovisual programs currently beingbroadcast (11-13), a recognition unit (8) for recognizing said semanticelements (41-43), which is linked to the detection set (7) and to atleast one program guide (20), said guide comprising for at least onebroadcasting channel, respectively at least one chronologically orderedlist (21-23) of information sets (21 ₁-21 _(n1), 22 ₁-22 _(n2), 23 ₁-23_(n3)) respectively associated with audiovisual programs intended to bebroadcast on said channel, said recognition unit (8) being capable ofselecting for each of said audiovisual programs currently beingbroadcast (11-13) on one of said channels, corresponding to one of saidordered lists (21-23), the one of said information sets (44-46) which ismost probably correlated with said semantic elements (41-43) obtained inrespect of said audiovisual program (11-13), and said recognition unit(8) being intended to perform a prior learning enabling it to selectsaid appropriate information sets (44-46) on the basis of said semanticelements (41-43), by means of learning materials comprising semanticelements and associated information, and a synthesis block (5) capableof synchronizing the audiovisual programs currently being broadcast(11-13) with the selected information sets (44-46).
 2. Thesynchronization device as claimed in claim 1, characterized in that:said device comprises a memory (6) designed to contain at least oneindicator (31-33) of the information set corresponding to theaudiovisual program broadcast last, respectively for each of said lists(21-23), the recognition unit (8) is capable of selecting saidinformation set (44-46) from said ordered list (21-23) for each of saidchannels, in such a way that said information set is positioned in saidordered list (21-22) at a rank higher than or equal to the rank of theinformation set pointed at by the indicator (31-33) associated with saidordered list (21-23), and the synthesis block (5) is intended forupdating said indicators (31-33) in the memory (6).
 3. Thesynchronization device as claimed in claim 2, characterized in that therecognition unit (8) is capable of selecting from the ordered lists(21-23), information sets (44-46) having a rank at least two unitshigher than the information sets (21 ₁-21 _(n1), 22 ₁-22 _(n2), 23 ₁-23_(n3)) pointed at by said indicators (31-33).
 4. The synchronizationdevice as claimed in any one of the preceding claims, characterized inthat the recognition unit (8) is intended also to perform said learningduring use.
 5. The synchronization device as claimed in any one of thepreceding claims, characterized in that said semantic elements (41-43)are chosen from: a family of shapes, a sound level, a pace of motion, afamily of visual and/or audible signs, and any combination of saidsemantic elements.
 6. The synchronization device as claimed in any oneof the preceding claims, characterized in that the recognition unit (8)comprises at least one neural network, intended for receiving as inputat least said semantic elements (41-43) and for producing as output atleast a result representative of information correlated with saidsemantic elements (41-43).
 7. The synchronization device as claimed inany one of the preceding claims, characterized in that said informationsets (21 ₁-21 _(n1), 22 ₁-22 _(n2), 23 ₁-23 _(n3)) comprise at least onevalue giving provisional time information regarding the broadcasting ofthe associated audiovisual programs.
 8. The application of thesynchronization device as claimed in any one of claims 1 to 7 to ascreen display of details regarding audiovisual programs currently beingbroadcast.
 9. The application of the synchronization device as claimedin any one of claims 1 to 7 to an automatic recording of audiovisualprograms by video recorder.
 10. The application of the synchronizationdevice as claimed in any one of claims 1 to 7 to a substitution ofcontent of audiovisual programs.
 11. A process for the synchronizationof broadcast audiovisual programs and of complementary information, inwhich: at least one semantic element (41-43) is extracted by means of apicture and/or sound detection set (7) from the content of audiovisualprograms currently being broadcast (11-13), from a chronologicallyordered list (21-23) of information sets (21 ₁-21 _(n1), 22 ₁-22 _(n2),23 ₁-23 _(n3)) respectively associated with audiovisual programsintended to be broadcast on a broadcasting channel is selected, by meansof a recognition unit (8), for each of said audiovisual programscurrently being broadcast (11-13) on said broadcasting channel, the oneof said information sets (44-46) which is most probably correlated withsaid semantic elements (41-43) obtained in respect of said audiovisualprogram (11-13), and the audiovisual programs currently being broadcast(11-13) are synchronized by means of a synthesis block (5) with theselected information sets (44-46), said synchronization process beingpreferably implemented by means of a synchronization device inaccordance with any one of claims 1 to 7.